Vermittlung von Effektgrößen an Lehrkräfte

Untersuchung verschiedener Visualisierungen

Jürgen Schneider
Kirstin Schmidt, Kristina Bohrer, Samuel Merk

18 Oct 2023

Kontext


  • “Clearinghouse” Ansätze
    Evidenzbasis systematischer Forschung bereitstellen (Knogleretal.2022?)



  • Verwendung der Materialien in Lehrpersonenbildung (Shavelson, 2020)



  • Effektstärken (ES) als eine der Schlüsselinformationen (Burns et al., 2011)



  • Wissenschaftler*innen und “Clearinghouses” verwenden in der Regel standardisierte textuelle ES-Metriken (Cohen, 1988)



educationendowmentfoundation.org.uk

clearinghouse.edu.tum.de

Warum Visualisierungen?




educationendowmentfoundation.org.uk

clearinghouse.edu.tum.de

Theorie

Lehrpersonen-orientierte Wissenschaftskommunikation






kognitive Verarbeitung


Verständnis


Relevanzwahrnehmung für die Praxis


(Jensen & Gerber, 2020)

Theorie

Lehrpersonen-orientierte Wissenschaftskommunikation






kognitive Verarbeitung
- Aufgabenschwierigkeit
- Effizienz
Verständnis


Relevanzwahrnehmung für die Praxis


(Korbach et al., 2017; Marcus et al., 1996)

Theorie

Lehrpersonen-orientierte Wissenschaftskommunikation






kognitive Verarbeitung
- Aufgabenschwierigkeit
- Effizienz
Verständnis
- Akkuratesse
- Sensitivität
Relevanzwahrnehmung für die Praxis


(Merk et al., in press)

Theorie

Lehrpersonen-orientierte Wissenschaftskommunikation






kognitive Verarbeitung
- Aufgabenschwierigkeit
- Effizienz
Verständnis
- Akkuratesse
- Sensitivität
Relevanzwahrnehmung für die Praxis
- Wahrgenommene Informativität
- Wahrgenommener Wert

(lortie-forguesetal.2021?)

Theorie

Forschungsinteresse & Studien

Delphi-Studie

Expertenurteil zu Lehrpersonen-orientierten Visualisierungen

explorativ
Studie 1

Vergleich der Wirkung verschiedener
Visualisierungstypen

explorativ
Studie 2

Vergleich der Wirkung verschiedener
Anreicherungsoptionen

konfirmativ

Delphi Studie

  • 4 Experten der Datenvisualisierung, 4 Experten der Wissenschaftskommunikation in Clearinghouses & Transfer

Delphi Studie

  • 4 Experten der Datenvisualisierung, 4 Experten der Wissenschaftskommunikation in Clearinghouses & Transfer



Ergebnisse: “Top ranked” Visualisierungen

Studie 1: Visualisierungstypen

Design

Studie 1

Vergleich der Wirkung verschiedener
Visualisierungstypen

explorativ
  • Lehrpersonen (N = 40; bayesian Updating)

  • 4 x 6 within-design
    • 4 Visualisierungstypen
    • 6 ES (d= -.8 to .8)

  • Randomisierungen
    • Reihenfolge der Bedingungen
    • Vignetten (1 von 4 between randomisiert)



Open Materials: github.com/j-5chneider/effsize_public

Studie 1: Visualisierungstypen

Instrumente

Perceived task difficulty How difficult was it for you to understand the figure? (Marcus et al., 1996)
Efficiency [time taken to answer sensitivity and accuracy] own creation
Sensitivity Is one group superior to the other or are they approximately the same? (Merk et al., in press)
Accuracy
...abstract metric The group that reads on... tablet is entirely superior to the one with paper - paper is entirely superior to the one with tablet own creation
...Cohen's U₃ Look at the mean test score of the group reading on paper: What percentage of the group that reads on tablet has a higher test score than this value? (Grice & Barrett, 2014)
...overlap How much percent of the groups will overlap on the test score? own creation
Perceived informativity How informative do you perceive the way the information is presented in the figure? (Lortie-Forgues et al., 2021)
Perceived value To what extent are these results relevant for your future teaching? own creation

Demo-Sruvey: es-vis-demo1.formr.org

Studie 1: Visualisierungstypen

Ergebnisse

Bayesianische Mehrebenenanalyse: Dummycodierte Visualisierungstypen


Visualisierungstypen haben Einfluss auf…

  • Aufgabenschwierigkeit:
  • Effizienz:
  • Akkuratesse:
    • Abstract metric:
    • Overlap:
    • Cohen’s U3:
  • Sensitivität:
  • Informativität:
  • Wert:

\((BF_{10} > 100)\)
\((BF_{10} > 100)\)


\((BF_{10} < 1/100)\)
\((BF_{10} > 100)\)
\((BF_{10} < 1/100)\)

\((BF_{10} = 16.62)\)
\((BF_{10} > 100)\)
\((BF_{10} > 100)\)

Open Data & Open Code:
github.com/j-5chneider/effsize_public

Studie 1: Visualisierungstypen

Ergebnisse




Type Task Difficulty Efficiency Accuracy overlap Sensitivity Informativity Value
Gardner-Altman (x-axis) 3.587 23379.21 0.024 0.658 3.800 3.800
Halfeye (x-axis) 4.233 17178.32 0.035 0.717 4.188 4.312
Halfeye (y-axis) 4.554 16889.87 0.001 0.723 4.404 4.383
Raincloud (y-axis) 3.829 20294.13 0.019 0.631 4.042 4.029

Studie 1: Visualisierungstypen

Ergebnisse

Diskussion

  • Visualisierungstyp relevanter Prädiktor erfolgreicher Kommunikation von Effektstärken


  • Halfeye-Plot als vielversprechend


  • Durchaus Gefahr durch Fehlkonzepote


  • nicht alleinstehend: Kontext Clearinghouses und Aus-/Fortbildung von Lehrpersonen

Vielen Dank



Jürgen Schneider
ju.schneider@dipf.de


Kooperation


Baird, M. D., & Pane, J. F. (2019). Translating Standardized Effects of Education Programs Into More Interpretable Metrics. Educational Researcher, 48(4), 217–228. https://doi.org/10.3102/0013189X19848729
Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011). The levels of evidence and their role in evidence-based medicine. Plastic and Reconstructive Surgery, 128(1), 305–310. https://doi.org/10.1097/PRS.0b013e318219c171
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Taylor and Francis.
Franconeri, S. L., Padilla, L. M., Shah, P., Zacks, J. M., & Hullman, J. (2021). The Science of Visual Data Communication: What Works. Psychological Science in the Public Interest, 22(3), 110–161. https://doi.org/10.1177/15291006211051956
Hanel, P. H. P., Maio, G. R., & Manstead, A. S. R. (2019). A new way to look at the data: Similarities between groups of people are large and important. Journal of Personality and Social Psychology, 116(4), 541–562. https://doi.org/10.1037/pspi0000154
Hanel, P. H. P., & Mehler, D. M. (2019). Beyond reporting statistical significance: Identifying informative effect sizes to improve scientific communication. Public Understanding of Science, 28(4), 468–485. https://doi.org/10.1177/0963662519834193
Jensen, E. A., & Gerber, A. (2020). Evidence-Based Science Communication. Frontiers in Communication, 4, 78. https://doi.org/10.3389/fcomm.2019.00078
Kim, Y.-S., Hofman, J. M., & Goldstein, D. G. (2022). Putting scientific results in perspective: Improving the communication of standardized effect sizes. CHI Conference on Human Factors in Computing Systems, 1–14. https://doi.org/10.1145/3491102.3502053
Korbach, A., Brünken, R., & Park, B. (2017). Measurement of cognitive load in multimedia learning: A comparison of different objective measures. Instructional Science, 45(4), 515–536. https://doi.org/10.1007/s11251-017-9413-5
Lipsey, M. W., Puzio, K., Yun, C., Herbert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., & Busick, M. D. (2012). Translating the Statistical Representation of the Effects of Education Interventions Into More Readily Interpretable Forms (Institute of Education Sciences, Ed.).
Lortie-Forgues, H., Sio, U. N., & Inglis, M. (2021). How Should Educational Effects Be Communicated to Teachers? Educational Researcher, 0013189X2098785. https://doi.org/10.3102/0013189X20987856
Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psychology, 88(1), 49–63. https://doi.org/10.1037/0022-0663.88.1.49
Shavelson, R. J. (2020). Research on teaching and the education of teachers: Brokering the gap. 17.

 

Icons:

Icons by Font Awesome CC BY 4.0

Theorie | Warum Wissenschaftskommunikation?

Theorie

Forschungsstand zur Visualisierung statistischer Information (für Laien)

Theorie | What we know about visualizing data (in general)

Theorie

Forschungsstand zur Visualisierung statistischer Information (für Laien)


Akkuratesse der Einschätzung statistischer Informationen: Visualisierungstyp spielt eine Rolle

. . .



Unterstützung im Prozess: Anreicherungsoptionen

Delphi study

  • 4 experts in data visualization, 4 experts in science communication in clearinghouses & transfer
  • phase 1: collection of 16 visualization types
    (for group values on a metric variable)

Delphi study

  • 4 experts in data visualization, 4 experts in science communication in clearinghouses & transfer
  • phase 1: collection of 16 visualization types
    (for group values on a metric variable)
  • phase 2: Rating and Ranking of 44 plots

“How accurately might teachers assess the ES depicted in the plot above?”
(7-point Likert scale; totally random - totally accurate)



Results: “Top ranked” visualizations

Study 1 | plot types: Results, Descriptives

Study 1 | plot types: Results efficiency

Study 1 | plot types: Results, Descriptives

Study 1 | plot types: Results, Descriptives

Study 1 | plot types: Results accuracy

Study 1 | plot types: Results accuracy

Study 1 | plot types: Results accuracy

Study 1 | plot types: Results accuracy

Study 1 | plot types: Results accuracy


without misconceptions

Study 2: Enrichment options

Design

Study 2

Comparison of the effect of different enrichment options on understanding.

confirmatory
  • Teachers (N = Bayesian updating)

  • 2 RCTs
    • Factor: visual benchmarking (yes vs. no)
    • Factor: signaling (difference, overlap, no signaling)

Study 2: Enrichment options

Design

  • Teachers (N = Bayesian updating)

  • 2 RCTs
    • Factor: visual benchmarking (yes vs. no)
    • Factor: signaling (difference, overlap, no signaling)

Study 2: Enrichment options

Design


  • Teachers (N = Bayesian updating)

  • 2 RCTs
    • Factor: visual benchmarking (yes vs. no)
    • Factor: signaling (difference, overlap, no signaling)

Study 2: Enrichment options

Design

  • increases accuracy (kimetal.2022?, Schmidt et al., 2023).
  • increases task difficulty (baddeley.1992?)
  • decreases effciency
  • increases informativeness
  • increases value

Studie 2: Anreicherungsoptionen

Design


  • increases accuracy (if no misconception)
  • reduces number of misconceptions
  • increases sensitivity
  • increases task difficulty
  • increases efficiency
  • increases informativeness
  • increases value

Folien: bit.ly/gebf23-v (view) | bit.ly/gebf23-g (download from github)